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Summary 


A  first  laboratory  version  of  an  Alertness  Monitoring/Management  (AMM)  system  has  been 
designed  and  implemented.  The  system  continually  estimates  the  level  of  alertness  of  a  human 
subject  using  EEG  spectral  information  recorded  from  the  subject’s  scalp,  and  delivers  auditory 
feedback  to  assist  the  subject  in  managing  his  or  her  own  level  of  alertness  in  work  environments 
requiring  constant  vigilance.  The  system  allows  experimenters  to  monitor  its  input  and  output 
via  real-time  color  graphics  displays. 

As  a  first  demonstration  and  evaluation  of  the  system,  six  subjects  participated  in  five  half- 
hour  sessions  (three  training  and  two  feedback  sessions),  which  involved  dual  detection  tasks 
simulating  the  passive  sonar  environment.  Auditory  targets,  3(X)-ms  noisebursts  presented  at  6 
dB  above  a  noise  background,  were  presented  at  a  mean  rate  of  10  targets  per  minute.  A  continu¬ 
ous  visual  waterfall  display  presented  illuminated  vertical  line  targets  at  a  mean  rate  of  one  per 
minute.  Subjects  pressed  one  response  button  to  report  noisebursts  and  another  to  report  visual 
targets. 

Neural  net  estimation  algorithms  were  trained  for  each  subject  to  estimate  the  current  pro¬ 
bability  of  detecting  auditory  targets  using  electroencephalogram  (EEG)  and  performance  data 
collected  during  one  or  more  of  the  initial  training  sessions.  During  feedback  sessions,  real-time 
signal  processing  and  individualized  neural  network  analysis  of  EEG  recorded  from  a  cenual 
scalp  electrode  were  used  to  estimate  continuously,  in  near  real-time,  the  current  probability-of- 
detection  of  auditory  targets.  Whenever  this  probabiliiy-of-detcclion  measure  declined  below  a 
preset  threshold  (e.g.,  when  it  predicted  more  than  a  40%  chance  of  failure  to  detect  the  auditory 
targets),  the  system  sounded  an  alarm  in  the  subject's  headphones.  When  training  sessions 
comprising  a  relatively  wide  range  of  detection  rates  were  used  to  train  the  estimation  algo¬ 
rithms,  the  alertness  esdroates  followed  changes  in  observed  detection  probability  relatively 
accurately. 

Four  of  the  six  subjects  reported  that  the  alertness  feedback  helped  them  to  maintain  detection 
performance.  A  fifth  subject  did  not  produce  enough  detection  lapses  to  fairly  evaluate  the  sys¬ 
tem.  Review  of  data  from  the  sixth  subject  suggested  that  future  versions  of  the  system  may  be 
able  to  provide  useful  feedback  to  this  subject  as  well.  Review  of  results  of  the  demonstration 
experiment  have  suggested  several  improvements  to  signal  processing  and  training  procedures 
used  in  the  system.  Effects  of  these  enhancements  on  system  performance  are  being  evaluated. 
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Introduction 


The  increasing  speed  and  complexity  of  current  information  technology  make  it  possible 
for  computerized  information  systems  to  monitor  the  physical  environment  with  increasing  acu¬ 
ity.  However,  the  capacity  of  the  human  operator  or  decision  maker  to  maintain  alertness  to  this 
information  is  not  increasing,  and  relatively  little  effort  has  been  invested  in  designing  technol¬ 
ogy  to  enhance  operators’  abilities  to  maintain  alertness.  Despite  common  assumptions  that 
while  we  are  awake  our  alermess  to  the  environment  is  secure  and  unbroken,  many  years  of  vigi¬ 
lance  research  attests  that  for  most  or  all  monitoring  equipment  operators  (e.g.,  air  traffic  control, 
sonar,  radar),  maintaining  a  constant  level  of  alertness  is  rare,  if  not  impossible  (Mackworth, 
1948).  Yet,  no  system  currently  exists  that  can  monitor  an  operator’s  level  of  alertness  directly, 
deliver  timely  information  about  lapses  in  alertness  to  the  operator  and/or  supervisor,  and  initiate 
appropriate  countermeasures. 

A  system  for  accurately  monitoring  operator  alertness  would  provide  several  benefits  that 
could  be  realized  in  a  variety  of  operational  environments.  The  direct  benefit  of  an  alertness 
monitoring  system  would  be  the  ability  to  deliver  early  warning  of  lapses  in  alertness  to  the 
operator  and/or  supervisor,  prompting  interventions  or  countermeasures  leading  to  a  direct 
improvement  in  total  man/machine  system  performance.  This  in  turn  could  reduce  or  eliminate 
the  need  for  operator  redundancy,  resulting  in  more  efficient  operation  and  substantial  cost  sav¬ 
ings  through  manpower  reduction.  Currently,  little  exact  information  is  available  on  the  dynam¬ 
ics  of  alenness  on  a  minute-by-minute  basis  for  sonar  and  radar  operators,  air  traffic  and 
engineering  plant  concrollers,  pilots,  and  workers  in  other  job  categories  requiring  constant  alert¬ 
ness.  An  alertness  monitoring  system  would  also  make  possible  direct  estimates  of  risk  of  system 
failure  due  to  loss  of  vigilance.  This  information  would  allow  more  productive  and  cost-effective 
human  engineering  and  staffing  designs  for  complex  systems,  which  could  result  in  additional 
overall  cost  savings. 

Since  the  landmark  investigations  of  Mackworth  (1948),  studies  of  alertness  have 
confirmed  that,  on  average,  detection  rates  in  laboratory  tests  begin  to  degrade  after  2  to  3  min  of 
task  performance,  eventually  reaching  a  plateau  during  which  70%  to  80%  of  targets  arc 
detected  (Makeig,  Elliott.  Inlow,  &  Kobus.  1990).  While  most  vigilance  research  in  the  past  has 
focused  on  measuring  mean  trends  in  performance,  performance  actually  tends  to  fluctuate  irreg¬ 
ularly  within  sessions,  with  periods  of  several  minutes  of  complete  unrcsponsivcncss  not  uncom¬ 
monly  observed  (Makeig  ei  al.,  1990).  During  the  last  century,  scattered  studies  have  presented 
evidence  for  the  presence  of  regular  fluctuattons  in  performance  at  cycle  lengths  ranging  from  a 
few  seconds  to  several  minutes  for  measures  including  time  estimation,  threshold  detection, 
reaction  time,  and  spontaneous  speaking  rate  (Stroud,  1966).  However,  most  previous  vigilance 
research  has  focused  on  simulating  actual  work  environments  using  relatively  low  target  presen¬ 
tation  rates  (2/min  or  less)  too  low  to  record  minute  scale  fluctuations. 

Methods  of  real-time  alertness  monitoring  are  needed  which  can  detect  the  actual  time 
courrc  of  performance  lapses.  The  only  direct,  noninvasive  measure  of  brain  state  available  for 
continuous  monitoring  in  an  operational  environment  is  the  recording  of  the  brain's  electromag¬ 
netic  activity  through  changes  in  potential  difference  between  points  on  the  scalp.  This  is  known 
as  the  electroencephalogram  (EEG).  Since  the  first  development  of  amplifiers  capable  of 
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resolving  brain  electrical  activity,  it  has  been  known  that  EEG  dynamics  change  dramatically 
during  and  near  periods  of  sleep  or  drowsiness  (Beatty,  Greenberg,  Deibler,  &  O’Hara,  1974). 
However,  in  the  past  there  have  been  several  obstacles  to  using  this  information  to  estimate 
effectively  and  in  near-real  time  an  operator’s  state  of  alertness.  Before  the  era  of  high-speed, 
portable  computing  and  appropriate  signal  processing  algorithms,  the  complexity  of  EEG 
dynamics  exceeded  the  ability  of  technology  to  extract  useful  information  in  real-time  for  use  in 
controlling  system  feedback  or  control. 

The  first  laboratory  version  of  an  Alertness  Monitoring/Management  (AMM)  system, 
described  here  has  several  advantages  over  previously  proposed  methods  of  objective  alertness 
estimation: 

1.  Many  proposals  for  performing  cognitive  monitoring  using  EEG  data  have  relied 
exclusively  on  timelocked  averaging  of  responses  to  unreliable  system  signals  (auditory  or 
visual  events)  or  to  signals  injected  into  the  operator’s  environment  as  probes  of  the 
operator’s  ability  to  respond.  These  methods  require  the  operator  to  frequently  shift  his  or 
her  attention  from  the  primary  monitoring  task.  It  would  be  preferable  to  monitor  operator 
state  without  introducing  distracting  tasks  or  signals  into  the  work  environment. 

2.  Previous  psychophysiological  studies  of  alertness  have  often  used  the  same  estimator  for  all 
subjects.  However,  the  relatively  large  individual  variability  in  EEG  dynamics  accompany¬ 
ing  loss  of  alertness  means  that  for  many  operators,  group  statistics  cannot  be  used  to  accu¬ 
rately  predict  changes  in  alertness  and  performance  (Makcig  et  al.,  1990;  Makeig  &  Inlow, 
1993),  and  estimators  customized  to  individual  operators  would  be  more  accurate  in  many 
cases. 

3.  Ffrevious  research  on  the  relationship  between  the  EEG  spectrum  and  alertness  used 
predefined  frequency  bands,  and  therefore  were  insensitive  to  individual  subject 
frequency-band  differences  (Beatty  et  al.,  1974:  Matousck  &  Peterson,  1983;  Belyavin  & 
Wright,  1987).  The  system  discussed  here  incorporates  a  more  flexible  signal  processing 
and  estimation  procedure  based  on  observations  of  changes  in  the  individual  EEG  spec¬ 
trum. 


Phase  I  •  System  Design  and  Realizatitm 

The  demonstration  project  consisted  of  two  pha»ss.  In  the  first  phase,  a  working  laboratory  ver¬ 
sion  of  a  real-time  AMM  system  was  designed  and  implemented.  This  involved  several  types  of 
real-dme  programming  of  three  Unix  systems  linked  by  Ethernet,  and  a  DOS-based  personal 
computer  with  graphics  display  capabilities.  A  schematic  diagram  of  this  system  is  shown  in 
Figure  1. 

The  signal  processing  pcrfomicd  by  the  system  has  several  stages: 

A.  Simultaneous  data  collection  and  dual-task  stimulation.  The  system  records  data  in  parallel 
on  two  ConcuiTcnt  (Real  Time  Unix)  workstations,  one  dedicated  to  auditwy  stimulus  syn¬ 
thesis  and  data  recording,  the  other  to  real-time  spectral  and  neural  net  analysis,  graphic 
display,  and  producing  auditory  feedback.  Another  program,  originally  developed  in  the 
laboratory  of  Dr.  Steven  Hillyard  at  the  University  of  California  at  San  Diego  (Hillyard  & 
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Figure  i.  Schematic  diagram  of  the  first  laboratory  version  of  an  Alertness 
Monitoring/Management  (AMM)  system  constructed  during  the  project.  The  subject 
(upper  right)  views  a  CRT  waterfall  display  while  hearing  sounds  through  headphones 
coming  from  a  sound  mixer,  and  presses  auditory  and  vi^al  response  buttons  (upper 
middle)  whenever  he  or  she  detects  target  signals  in  the  respective  modalities.  Auditory 
signals  include  a  noise  background,  tones,  and  noiseburst  targets  synthesized  by  a 
Concurrent  workstation  (center  tefi)^  and  occasional  auditory  alertness  feedback  signals 
(brief  buzzes)  synthesized  by  a  second  Concurrent  workstation  (center).  The  first 
Concurrent  machine  also  records  EEG  and  performance  information  to  disk.  The  second 
Concurrent  machine  processes  EEG  information  in  near-real  time,  creates  an  climate  of 
the  subject’s  current  probability  of  failing  to  respond  to  the  noiseburst  signals,  and 
produMS  auditory  feedback  signals  whenever  the  estimated  error  probability  rises  above  a 
preset  threshold.  The  Concurrent  mar;*lrc<f-'  linked  to  a  higher-speed  MIPS 
workstation  (upper  left)  for  off-line  data  analysis.  A  video  camera  and  recorder  (upper  right) 
record  video  and  auditory  stimulus  information  during  the  experiments. 


Johnston,  submitted),  runs  on  a  dedicated  PC.  It  presents  visual  stimuli  against  a  random 
visual  waterfall  background,  and  delivers  stimulus  timing  information  to  the  Concurrent 
computers. 

B.  Data  selection,  scaling,  and  artifact  rejection.  Real-time  artifact  rejection  and  scaling  rou¬ 
tines  preprocess  the  EEC  data  using  coefficients  read  from  a  file  created  by  the  neural  net 
training  software.  Data  from  epochs  contaminated  by  artifacts  are  replaced  by  the  most 
recent  accepted  values. 

C.  Robust  real-time  spectral  estimation.  Real-time  spectral  analysis  of  the  data  is  accom¬ 
plished  using  Hanning-windowed  data  windows  of  256  points  with  50%  overlap.  After 
rejecting  windows  in  which  data  exceed  +/-50  uV,  median  filtering,  a  robust  statistical 
smoothing  technique,  is  used  on  the  estimates,  to  further  minimize  the  presence  of  data 
artifacts,  principally  muscle  noise,  in  the  EEG  record.  The  moving  median  filter  uses  a 
moving  square  window  of  1 1  overlapping  power  estimates  for  each  frequency. 

D.  Off-line  neural  net  training.  An  automated  search  procedure  selects  optimum  EEG  fre¬ 
quencies  and  time-smoothing  parameters  for  each  subject.  A  two-layer  perccptron  network 
is  then  trained  to  use  smoothed  power  estimates  at  these  frequencies  to  optimally  predict 
local  error  rate,  defined  as  the  percentage  of  targets  not  responded  to  (i.c.,  lapses)  in  a  mov¬ 
ing  time  window,  A  95-s  square  window  is  advanced  through  the  performance  record  in 
steps  of  1.64  s  to  create  the  local  error  rate  measure  from  the  sequence  of  behavioral 
responses  (hits  and  lapses).  Inicrtarget  intervals  arc  randomized  in  order  to  prevent  per¬ 
severation  in  subject  responses,  and  therefore  smoothing  requires  use  of  a  specially 
designed  smoothing  program  that  produces  smoothed,  regularly  spaced  estimates  from 
irregularly  spaced  data. 

Inputs  to  the  neural  network  estimation  algorithm  consist  of  smoothed  and  median-filtered 
EEG  power  estimates  at  a  set  of  five  EEG  frequencies.  These  frequencies  arc  chosen  for 
each  subject  on  the  basis  of  previously  published  coherence  results  (Makcig  &.  Inlow, 
1993).  A  sigmoidal  activation  function  is  used  at  the  single  second-layer  node.  The  net¬ 
works  are  trained  using  a  standard  backpropagation  method  to  predict  the  smoothed  local 
error  rate  measure.  A  training  script  directs  a  commercial  neural  net  simulation  package 
(SN2)  to  train  the  perccptron  network.  We  have  recently  shown  that  such  networks  give 
better  estimation  performance  for  this  problem  than  linear  regression  (Venturini,  Lyiton, 
Sejnowski,  Inlow,  Elliott,  &.  Makeig,  submitted).  For  the  demonstration  expenn^ents,  this 
network  was  trained  on  data  from  the  first  sessions  of  each  subject,  producing  an  individu¬ 
alized  alertness  estintation  network  for  each  subject, 

£.  Real-tune  neural  network  estimation.  During  subsequent  feedback  sessions,  the  weights 
from  the  previously  trained  network  for  each  subject  were  read  into  a  teal-time  feed¬ 
forward  neural  network  module  that  computes  a  local  error  rate  estimate  from  the  smoothed 
real-time  spectral  estimates  every  1.64  s. 

F.  Auditory  feedback.  During  feedback  sessions,  whenever  estimated  error  rate  rises  above  a 
preset  level  (e.g.,  40%),  the  system  synthesizes  a  1-s  burst  of  square-wave  sound  using  the 
clock  board  of  the  Concurrent  computer.  An  analog  sound  mixing  board  then  feeds  this 
signal  into  the  subject’s  headphones.  The  ahum  repeats  once  each  1.64-s  epoch  during 


which  the  error  rate  estimate  remains  above  the  alarm  threshold.  To  maximize  alarm 
impact  on  the  subject,  if  the  alarm  sounds  for  more  than  8  epochs  in  a  row  it  is  disabled  for 
another  16  epochs,  unless  the  error  rate  estimate  first  dips  below,  then  rises  above  the  alarm 
threshold. 

F.  Graphic  display  of  the  estimation  process:  Real-time  graphic  displays  help  the  experi¬ 
menters  monitor  the  experiments,  and  provide  a  dynamic  means  of  demonstrating  the 
operation  of  the  system.  A  set  of  multiwindow  real-time  graphics  modules  display  the  raw 
incoming  EEG  and  the  resulting  power  spectral  values  input  to  the  estimation  network. 
These  spectral  values  are  overplotted  on  a  plot  of  the  correlations  between  power  at  each 
EEG  frequency  and  performance,  during  the  training  experiment  used  to  train  the  network. 


Phase  II  •  Demonstration  and  Evaluation 

As  a  first  demonstration  and  evaluation  of  the  system,  30  experimental  sessions  were  conducted. 
In  some  of  the  sessions,  real-time  auditory  feedback  was  delivered  to  the  subject.  These  experi¬ 
ments  were  the  first  of  an  continuing  series  whose  goal  is  to  enhance  and  evaluate  the  operation 
of  the  prototype  AMM  system. 


Methods 

Subjects 

Six  volunteers,  three  male  and  three  female,  participated  in  dual-task  auditory  and  visual 
simulations  of  a  passive  sonar  target  dcicaion  environment.  Two  of  the  subjects  were  prospec¬ 
tive  students  in  a  Navy  sonar  course.  The  remaining  subjects  were  members  of  the  laboratory 
staff.  Ages  ranged  from  18  to  36  years  (M  =  24.5  years).  None  of  the  subjects  reported  any 
known  hearing  loss;  the  three  males  had  all  passed  standard  Navy  hearing  tests. 

Stimuli 

A  Orncurrent  Real  Time  Unix  computer  system  synthesized  the  auditory  stimuli  using  a 
16-bit  D/A  convener  sampling  at  50  kHz.  The  auditory  stimulus  streams  used  in  the  experiment 
arc  shown  schematically  in  Figure  2.  The  stimuli  were  presented  binaurally,  in  a  white  noise 
background  at  63  dB  (A-weighting),  through  headphones  mounted  in  isolating  cuffs.  For  con¬ 
tinuity  with  the  existing  experimental  design,  task-irrelevant  probe  tones  of  two  frequencies  (568 
Hz  and  1098  Hz)  were  presented  in  random  order  with  interstimulus  intervals  of  2  s  to  4  s  at  72 
dB.  Probe  tones  were  50  ms  in  duration,  with  rise  and  fall  times  of  10  ms.  High  tones  were 
presented  more  frequently  (probability  of  occurrence  80%)  than  were  low  tones  (20%). 

Audittay  targets  were  30O-ms  noisebursis  with  rise  and  fall  times  of  150  ms  and  110  ms, 
respectively.  Noisebursi  targets  occurred  in  50%  of  the  2-s  to  4-s  interstimulus  intervals  (i.e.,  at  a 
mean  rate  of  10/min)  and  were  presented  at  approximately  6  dB  above  the  noise  background. 
The  auditory  feedback  alarm  consisted  of  a  200-Hz  square  wave  tone.  800  ms  in  duration, 
presented  at  approximately  75  dB.  The  auditory  stimuli  were  similar  to  those  used  in  earlier 
alertness  monitoring  expeiiments  (Makeig  &  Inlow,  1993),  except  that  in  the  present 
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experiments,  auditory  steady-state  response  stimulation  was  not  used. 


TARGET  BURST 


1098  Hz  1098  Hz  568  Hz 


Figure  2.  Schematic  depiction  of  the  auditory  stimuli  used  in  the  experiments.  Frequent 
(80%)  and  Rare  (20%)  tones  were  presented  at  random  interstimulus  intervals  of  2  s  to  4  $. 
In  half  tile  intertone  intervals,  noiseburst  targets  were  presented  6  dB  above  their 
threshold  in  the  63  dB  white  noise  background,  which  was  continuously  presented  during 
the  experiments.  Auditory  alarm  signals  consisted  of  !•$,  75>dB,  200-Hz  square-wave 
stimuli.  During  feedback  sessions,  these  were  presented  every  1.64  s  that  (he  estimated 
error  rate  for  the  auditory  detection  task  rose  above  a  preset  threshold  (normally  40%). 


Visual  stimuii  arc  produced  using  a  PC  (Evercx  EX-3000R)  with  a  CTTX  14-inch,  VGA 
Color  Monitor  (CVP-5468).  The  display  was  13-cm  wide  by  9-cm  high.  The  display  background 
was  composed  of  1-mm  squares  filled  with  randomly  assigned  greyscale  values,  the  result 
resembling  visual  television  noise  (''snow").  This  display  was  updated  twice  each  second  by- 
adding  new  lines  of  squares  to  the  top  of  the  screen  and  scrolling  the  existing  display  down  one 
line,  creating  a  moving  "waterfall"  effect.  Visual  largcts.  introduce!  at  a  mean  rate  of  1/min, 
consisted  of  20  consecutive  white  squares  forming  a  vertical  line.  The  target  line  was  embedded 
in  the  ciuuer  of  the  display  background,  and  scrolled  down  the  display  freun  top  to  bottom  with 
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the  rest  of  the  display. 

Procedure 

Each  subject  participated  in  five  28-tnin  experimental  sessions  on  separate  days.  Subjects 
sat  in  a  chair  with  their  right  index  and  middle  fingers  resting  on  the  visual  and  auditory  target 
response  buttons,  respectively.  The  chair,  which  was  located  in  a  sound-resistant  booth,  was 
positioned  so  that  subjects  faced  a  39-by'48-cm  window  in  the  booth’s  wall.  The  CRT  monitor 
was  positioned  approximately  1 10  cm  from  the  subjects’  eyes,  so  the  display  background  sub¬ 
tended  approximately  7  degrees  of  visual  angle. 

At  the  beginning  of  each  session,  the  experimenter  read  a  set  of  standardized  instructions. 
Subjects  were  instructed  to  press  one  of  two  response  buttons  (auditory  or  visual)  whenever  they 
detected  an  auditory  or  visual  target.  The  subjects  then  were  given  from  2  min  to  5  min  of  train¬ 
ing  on  the  tasks  in  three  stages:  auditory  alone,  visual  alone,  and  auditory  plus  visual. 

The  first  three  sessions  for  each  subject  were  training  sessions.  The  estimation  network 
was  then  trained  on  EEC  spectral  and  perfonnance  data  from  the  training  session  that  contained 
the  most  failures  to  respond  (lapses).  The  training  procedure  generated  sets  of  coefficients  for  an 
error  estimation  algorithm  matching  individual  subjects’  EEC  dynamics.  In  subsequent  feedback 
sessions,  this  individualized  estimation  algorithm  was  applied  to  spontaneous  EEC  to  estimate 
changes  in  mean  current  error  rate  in  a  95-$  moving  exponential  time  window.  When  the  error 
estimate  exceeded  a  predetermined  threshold  level  (most  ofien  40%).  auditory  alarm  signals 
were  delivered  to  the  subject  via  the  subject’s  headphones. 

Data  Collection 

Data  from  all  sessions  were  continuously  recorded  to  disk  for  off-line  analysis.  Standard 
Crass  EEC  amplifiers  (bandpass  0.1-  to  lOO-Hz)  amplified  EEC.  electrooculogram  (ECX)),  and 
electrocardiogram  (ECO)  signals  SOk,  lOk,  and  3k  times,  respectively.  Biopotential  data  were 
multiplexed  with  response  button  press  information  and  converted  to  16-bit  digital  format  at  a 
sampling  rate  of  312.S  Hz.  EEC  data  were  recorded  from  two  midline  sites,  one  frontal  (Fz)  and 
the  other  midw'ay  between  parietal  and  occipital  sites  (Fz/Oz),  using  10-mm  gold-plated 
electrodes  referenced  to  the  right  earlobe.  Periocular  clcciixxlcs  were  used  to  record  electrical 
potentials  generated  by  eye  moventents.  One  electrode  was  placed  at  the  outer  canthus  of  the 
right  eye  and  another  about  0.5  cm  above  the  left  eyebrow.  The  ECX3  data  were  recorded  from  an 
electrode  placed  approximately  4  cm  below  the  left  clavicle.  Electrical  impedanos  at  ail 
electrode  sites  was  adjusted  to  less  than  S  kOhms  by  cleaning  the  skin  under  the  electrodes  with 
abrasive  gel.  Before  each  experiment,  the  EEC  amplifiers  were  calibrated  to  compensate  for  any 
differences  between  actual  and  nominal  gain  values. 

Analy^ 

Auditory  targets  were  classified  as  Hits  if  the  subject  pressed  the  Auditory  response  button 
within  120  ms  to  3000  ms  of  target  onset,  or  were  classified  as  Lapses  if  they  did  not.  Responses 
to  visual  targets  were  considered  Hits  if  the  subject  pressed  the  visual  target  button  while  the  tar¬ 
get  was  on  the  screen  (i.c..  within  20  s  after  it  began  to  appear).  Auditory  performance  was 
quantified  by  smoothing  with  a  9S-s  exponential  window  whose  gain  varied  smoothly  from  1.0 
at  the  leading  edge  to  0.1  at  the  trailing  edge.  Visual  performance  was  smoodted  with  a  15S-s 
bell-shaped  (Papoulis)  window  w  .'  ich  was  treated  as  non-causal  (i.e.,  ’’now’’  was  considered  to 


be  the  middle  of  the  visual  performance  window).  Smoothed  estimates  of  error  rates  and  EEG 
power  at  8 1  EEG  frequencies  were  produced  at  intervals  of  L64s. 

Moving-averaged  EEG  spectral  power  was  calculated  as  follows:  Hanning-windowed  data 
windows  of  256  points  with  50%  overlap  were  converted  to  the  frequency  domain  using  512- 
point  Fast  Fourier  Transforms  (FFTs).  Prior  to  transforming  to  spectral  power,  a  simple  out-of- 
bounds  test  rejected  data  epochs  containing  gross  muscle  artifacts.  Median  filtering  was  then 
performed  on  the  accepted  epochs  using  a  moving  window  of  1 1  overlapping  power  estimates  at 
each  frequency.  When  a  data  window  contained  one  or  more  out-of-bounds  value,  the  latest 
^epted  values  were  substituted  prior  to  median  filtering 

For  each  session,  power  at  each  EEG  frequency  was  correlated  with  changes  in  local  error 
rate  on  the  auditory  task.  To  optimize  this  correlation  for  the  purpose  of  real-time  alertness  esti¬ 
mation,  the  correlation  was  calculated  separately  for  spectral  power  values  smoothed  with  a  set 
of  95-s  exponential  windows  with  different  decay  rates.  The  decay  rate  giving  the  best  correla¬ 
tion  was  selected,  along  with  five  EEG  frequencies  for  use  in  the  estimation  network.  These 
were  the  most  highly  correlated  single  frequencies  in  each  of  five  frequency  bands  whose  power 
was  shown  in  earlier  experiments  (Makeig  &  Inlow,  1993)  to  be  most  sensitive  to  changes  in 
detection  performance. 

To  train  the  neural  network  for  each  subject,  EEG  power  from  a  training  session  (without 
feedback)  was  used  to  train  a  two-layer  perceptron  neural  network.  Data  from  each  session  con¬ 
sisted  of  1024  spectral  and  error  rate  estimates  at  1.64-s  intervals.  Of  these.  260  wcnc  selected 
for  testing  the  generalization  of  the  network  training  results  to  determine  a  stopping  point  for  the 
net  training.  EEG  power  values  were  normalized  for  import  to  the  network  by  linearly 
transforming  the  20ih  and  liOth  peiceniiles  of  the  power  distribution  in  the  training  session  to 
range  from  -l.Oto  1.0. 

In  subsequent  sessions,  to  compute  the  continuous  esUmate  of  local  error  rate,  power  at  the 
same  five  frequencies  used  to  train  the  network  was  first  smoothed  with  the  same  causal  95-s 
exponential  windows  selected  to  train  the  network,  normalized  using  the  same  scaling  parame¬ 
ters.  and  delivered  to  the  feedforward  neural  network  software. 

Feedback 

In  experimenis  employing  auditory  feedback,  the  local  erTx>r  rate  estimate  was  compared 
against  the  alarm  threshold  value.  In  most  of  the  feedback  sessions  this  threshold  was  set  to  40% 
estimated  errors.  If  the  estimate  exceeded  the  threshold,  a  0.8-s.  200-Hz  square  wave  burst  at  75 
dB  was  synthesized  by  the  c'^puter's  clock  board,  mixed  with  the  auditory  stimuli,  and 
delivered  to  the  subject.  To  avoid  nmnotony  and  substx^uent  loss  of  arou^l  value,  a  rule  was 
implemented  that  once  the  alarm  had  sounded  for  8  consecutive  epochs,  it  was  silenced  f(»r  16 
more  epochs  unless  the  alenness  estimate  fell  below  and  then  returned  above  its  threshold  value. 
Alarm  history  was  saved  for  later  review. 


Results 


Subject  Performance 

Local  error  rate,  the  performance  measure  used  in  these  experiments,  is  not  a  uniquely 
defined  measure.  In  the  summary  plots  below,  a  155-s  (95-epoch)  noncausal  bell-shaped 
Papoulis  window  was  used  to  compute  local  error  rate.  Total  percentage  of  correct  responses  to 
auditory  targets  in  the  20  experimental  sessions  ranged  from  99.6%  to  45.4%.  Mean  auditory 
error  rates  for  the  20  sessions,  and  visual  error  rates  for  18  sessions  (shown  in  Figures  3a  and  b), 
followed  the  same  trend  as  earlier  experiments,  with  group  mean  error  rate  increasing  for  the 
first  10  min  and  remaining  stable  thereafter.  (Visual  data  for  two  sessions  were  lost  because  of 
equipment  failure).  Although  earlier  experiments  had  revealed  a  tendency  for  a  monotonic 
increase  in  mean  error  rate  across  repeated  sessions,  in  these  sessions  no  such  trend  was 
observed. 

Note  the  strong  similarity  (r  =  0.73)  between  the  mean  auditory  and  visual  performance 
records.  The^  results  confirm  results  of  our  earlier  auditory  detection  experiments  (Makeig  et 
al.,  1990),  and  of  classical  vigilance  experiments  (Mackworth.  1948).  In  individual  sessions, 
however,  local  error  rates  fluctuated  widely,  as  shown  in  Figure  4,  in  which  each  local  error  rate 
trace  may  vary  from  a  low  of  0%  (all  correct)  to  a  high  of  100%  (no  responses).  Note  in  Figure 
4  that  the  four  subjects  have  different  mean  error  rates  and  no  consistent  pattern  of  fluctuations 
within  sessions.  On  average,  two  of  the  subjects  (dl  and  d2)  had  lower  mean  error  rates  than  ’’  c 
other  two  (d3  and  d4). 

Error  Rate  Estimation 

Figure  5  below  shows  estimated  and  actual  error  rates  for  one  feedback  session  each  from 
three  of  die  four  subjects.  A  qualitative  fit  between  the  estimated  and  actual  error  rates  is 
apparent.  In  each  case,  during  segments  in  which  estimated  error  rate  was  above  the  alarm  thres¬ 
hold,  local  error  rate  was  high  relative  to  the  rest  of  the  session. 
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Figure  3.  Group  means  In  auditory  and  visual  task  error  rates.  For  the  auditory  task, 
(doited  line)  average  of  20  sessions  on  four  subjects.  Visual  error  rate  data  (solid  line)  from  two 
sessions  was  lost  due  to  mechanical  difficulties,  so  the  visual  error  rate  curve  averages  data  from 
18  sessions.  These  curves  reproduce  those  found  in  earlier  experiments  (Makeig  &  Inlow,  1993), 
and  follow  trends  reported  in  previous  vigilance  research. 


Discussion 

These  experiments  have  demonstrated  the  feasibility  of  objective  real-time  alertness 
estimation  in  a  laboratory  AMM  system  by  delivering  auditory  alarm  signals  to  operators  when 
estimated  alertness  became  lower  than  a  preset  threshold.  Alertness  level  is  difficult  to  estimate 
subjectively,  since  loss  of  aleitness  is  accompanied  also  by  loss  of  self-awareness  and  memory. 
This  may  also  be  one  reason  that  individual  fluctuations  in  alertness  have  not  been  studied  much 
in  previous  vigilance  research.  Nevertheless,  at  the  end  of  the  three  sessions  shown  in  Figure  S, 
each  of  the  three  subjects  reported  that  the  auditory  feedback  helped  them  to  maintain  alenness 
by  giving  tnem  objective  confirmation  of  changes  in  their  levels  of  alertness. 

However,  differences  between  the  estimated  and  actual  error  rate  curves  remain  larger  than 
e'^pected  on  the  basis  of  Monte  Carlo  simulations  and  earlier  work  using  linear  regression  (Mak¬ 
eig  &  Inlow,  1993).  As  the  system  design  was  changed  during  the  sessions,  quantitative 
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Figure  -  L  *411  error  rate  time  series  for  individual  sessions.  Note  that  despite  the 
predictabiliiy  of  the  group  mean  rates  in  Figure  3,  individual  error  rate  dynamics  vary 
considerably.  The  three  starred  runs  are  examined  in  more  detail  in  Figure  5  belovir. 


performance  statistics  will  be  reponed  later,  after  a  larger  number  of  subjects  are  run  using  the 
same  version  of  the  system.  In  the  two  feedback  sessions  for  the  founh  subject,  the  error  rate 
estimate  did  not  follow  the  actual  enor  rate  or  subjective  alertness  changes  accurately  enough 
for  the  subject  to  report  that  feedback  had  been  useful  in  managing  his  alertness.  Recently,  we 
trained  a  network  using  Version  1.1  of  the  system  and  tested  this  net  on  data  fmm  this  subject’s 
other  four  sessions.  Results  suggest  that  Version  1.1  is  capable  of  producing  useful  alenness 
estimates  for  this  subject  Review  of  the  results  of  30  sessions  has  pncmipted  several  ideas  for 
improving  the  alertness  estimation  and  training  procedure: 

1.  In  Version  1.0  of  the  system,  which  was  tested  on  the  first  four  subjects  (Fig.  S),  the 
smoothing  window  used  to  compute  local  error  rate  and  train  the  neural  net  algorithm^^  con* 
sisted  of  a  two-sided  or  noncausal  bell-shaped  window.  In  real-time  spectral  estimation, 
however,  noncausal  hlicring  cannot  be  used,  because  it  requires  "future"  data  which 
become  available  only  after  an  unacceptable  delay.  In  the  real-time  system,  thcrefote,  a 
trailing  or  causal  exponential  window  was  used  to  smooth  the  specU'al  data.  In  system  Ver¬ 
sion  1.1,  which  so  far  has  been  tested  on  oniy  two  subjects  (figs.  6  and  7),  the  eirm-  rate  and 
EEG  windows  used  to  train  and  run  the  estimation  network  are  causal. 
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Figure  5.  Actual  and  estimated  local  error  rate  for  the  auditory  task  for  three  feedback 
sessions  on  three  of  the  four  subjects.  The  alarm  threshold  level  is  also  shown  on  each  plot 
(See  text  for  further  details). 
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2.  In  Version  1.0  of  the  system,  there  were  differences  between  the  artifact  rejection  and 
median  smoothing  in  the  computation  of  the  EEG  spectra  for  training  and  running  the  net¬ 
works.  In  Version  1.1,  both  algorithms  are  identical. 

3.  The  subjects  judged  the  feedback  alarm  signals  used  in  these  experiments  as  sufficiently 
arousing  on  first  presentation.  However,  in  Version  1.0  the  signals  tended  to  mask  auditory 
targets  that  were  sometimes  presented  simultaneously.  Therefore,  in  Version  1.1,  the  alarm 
signals  are  shortened.  In  future  versions  of  the  system,  the  alarms  should  increase  in  inten¬ 
sity  or  urgency  if  the  error  rate  estimate  remains  high.  The  subject  should  also  be  able  to 
control  the  volume  of  the  alarm.  Finally,  use  of  more  complex  auditory  feedback,  includ¬ 
ing  prerecorded  voice  messages,  should  be  explored.  In  more  advanced  versions  of  the  sys¬ 
tem,  continuous  visual  alertness  information  could  be  made  available  to  the  subject 

4.  It  may  be  that  improved  estimation  accuracy  can  be  attained  if  the  neural  network  used 
power  at  more  than  five  EEG  frequencies,  and/or  if  it  employed  a  more  complex  structure 
and  training  procedure.  Other  network  structures  and  training  procedures  are  being  investi¬ 
gated. 

5.  While  monitoring  from  a  single  scalp  channel  might  be  optimum  form  a  practical  point  of 
view,  it  is  likely  that  significant  improvements  in  statistical  power  and  noise  cancellation 
can  be  achieved  by  collecting  data  from  two  or  more  scalp  channels,  by  monitoring  evoked 
responses  to  the  task-irrelevant  probe  tones  (Makeig  et  al.,  1990)  and/or  by  monitoring 
other  psychophysiological  channels  (such  as  eye  movements  and  heart  rate).  Data  from 
multiple  physiological  information  channels  could  then  be  combined  in  a  more  advanced 
alertness  estimation  algorithm.  We  are  now  evaluating  these  possibilities. 

6.  *^arlier  research  (^  Makeig  &  Inlow,  1993)  showed  that  to  ensure  accurate  error  rate  estima¬ 
tion,  the  training  uata  must  cor  tain  a  wide  range  of  local  error  rates  For  at  least  two  sub- 
jcc,s  (d2  and  d4),  local  error  rate  remained  low  throughout  the  training  sessions  and  there¬ 
fore  could  not  be  used  to  train  the  estimation  network  Future  experiments  should  be  long 
enough,  and/or  should  inanipulate  subject  sleep,  caffeine  intake,  etc.,  so  as  to  maximize  tlie 
range  of  local  error  rates  in  the  training  sessions.  Optimum  methods  of  taking  advan¬ 
tage  of  partial  bctwcen-subject  similarities  to  combine  training  session  data  from  more  than 
one  subject  should  also  be  studied. 

Test  ot  First  Enhancements 

To  date,  Version  1.1  of  the  system  haS  been  tested  on  two  subjects.  The  fifth  subject,  a  profes¬ 
sional  sonar  operator,  did  not  prod*  "  enough  deteci'on  lapses  in  his  training  sessions  to  allow 
an  estimation  algorithm  to  be  trained  on  his  data.  Results  of  one  feedback  ru.  on  the  sixth  sub¬ 
ject  are  presented  in  Figures  6  and  7.  This  subject  was  also  a  sonar  operator  'but  unlike  the  pre¬ 
vious  subject,  not  a  corfee  drink''r).  Figure  6  shows  actual  and  estimated  error  rate  for  a  28-min 
session  with  alertness  feedback.  The  alarm  threshold  and  alarm  record  arc  also  shown,  along 
with  head  drop,  which  was  noted  from  the  videotape  taken  of  the  subject  duiing  the  session.  The 
estimation  algorithm  used  power  at  five  EEG  frequencies  chosen  and  smoothed  individually  for 
the  subject  on  the  basis  of  a  previous  training  session.  As  aiiticipated,  at  the  beginning  of  the  mn, 
the  estimation  algorithm  needed  95  s  of  data  to  begin  making  accurate  error  rate  estimates. 
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In  Figure  6,  note  the  following  points:  Aftei  the  session,  the  subject  resported  that  the  early 
alarm  (at  minute  2)  was  appropriate,  since  he  was  feeling  drowsy  at  the  time,  and  actually 
missed  one  auditoiy  target.  The  subject  stated  that  the  alarm  reminded  him  to  be  more  attentive. 
The  error  rate  estimate  began  to  rise  in  minutes  5  to  9,  i.e.  about  4  minutes  before  the  subject  had 
another  response  lapse.  Head  drop  became  visible  almost  12  min  after  the  alarm  threshold  was 
reached  by  the  error  rate  estimate  (at  minute  10),  and  the  subject  began  making  errors.  The  esti¬ 
mation  algorithm  accurately  estimated  the  subject’s  return  to  alertness  at  minute  19,  which  was 
aided  by  an  unplanned  verbal  query  to  the  subject  from  the  experimenter,  who  thought  the  sound 
delivery  system  might  be  malfunctioning.  Error  rate  during  the  final  portion  of  the  run  (minutes 
20  to  28)  was  accurately  tracked  by  the  system  without  use  of  any  information  except  the 
subject’s  EEG  spectrum. 

Figure  7  shows  a  post  hoc  alertness  estimate  of  error  rate  in  this  same  session,  computed 
from  EEG  power  estimates  at  the  same  five  frequencies  and  using  the  same  smoothing  constants 
as  were  used  in  the  real-time  algorithm  (Fig.  6).  The  same  training  session  was  also  used  to  train 
the  revised  estimation  net.  In  the  revised  net,  however,  normalization  constants  were  based  on 
two  train!  ig  sessions  rather  than  one.  In  Figure  7,  note  that  the  root  mean  square  (RMS)  error 
between  tlie  estimated  and  observed  error  rates  (0.08)  is  less  than  that  observed  in  Monte  Carlo 
simulation  >  using  a  predetermined  error  rate  probability  curve  to  generate  stochastic  scries  of 
Hits  and  Lapses  with  the  same  temporal  statistics  as  in  the  actual  experiments.  Hence,  statisti¬ 
cally,  the  prediction  error  in  Figure  7  is  as  small  a  possible,  and  further  improvement  in  estima¬ 
tion  accuracy  cannot  be  expected.  The  extremely  close  fit  of  the  estimated  and  actual  error  rate 
record  in  Figure  7  demonstrates  that  precise  information  about  the  exact  alcnness  trend  is  con¬ 
tained  in  the  EEG  power  spectrum,  as  reported  by  Makeig  and  Inlow  (1993).  However,  power 
normalization  remains  an  important  area  for  further  research. 
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Figure  6.  Results  of  a  session  on  a  fifth  subject  employing  reahtime  alertness  monitoring 
and  feedback.  In  this  experimenty  the  feedback  alarm  threshold  was  set  at  0.4.  The  actual 
alarm  signal  record  is  shown  in  the  lowest  trace.  The  square  pulses  In  the  dashed  trace 
above  it  indicate  moments  when  the  subject's  head  was  observed  to  drop  in  the  videotape 
of  the  session.  The  dark  trace  shows  the  actual  local  error  rate  on  the  auditory  detection 
task,  smoothed  using  a  95>s  exponential  window  whose  center  of  gravity  trailed  the  leading 
edge  of  the  window  by  22  s.  The  top  trace  shows  the  error  rate  estimate  computed  fh)m  the 
subject's  EEG  spectrum  during  the  run. 
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Figure  7.  This  figure  demonstrates  that  significant  improvement  in  estimation  accuracy 
would  have  been  obtained  had  the  normalisation  constants  for  iMreprocessing  the  smoothed 
EEC  power  been  computed  using  data  firom  two  training  runs  rather  than  from  just  one. 
Hie  figure  shows  post  hoc  error  rate  estimation  using  the  same  EEG  data  as  in  Figure  6, 
but  with  revised  normalization  parameters  computed  using  Version  1.1  of  the  training 
algorithm  on  the  basis  of  two  previous  training  runs  on  this  subject.  Estimation  accuracy 
for  this  session  cannot  be  distinguished  from  measurement  error  for  the  local  error  rate 
measure. 
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Conclusions 


First  laboratory  versions  of  an  individualized  Alertness  Monitoring/Management  (AMM) 
system  have  been  demonstrated  and  have  been  tested,  to  date  (January,  1993),  in  a  total  of  30 
half-hour  test  sessions  on  six  subjects.  Results  of  these  experiments  suggest  that  sufficient  infor¬ 
mation  is  available  in  the  EEG  spectrum  recorded  from  a  single  scalp  derivation  to  accurately 
estimate  probability  of  detection  in  an  auditory  detection  task  performed  by  subjects  con¬ 
currently  with  a  visual  task  simulating  one  aspect  of  sonar  operations.  Future  research  will 
attempt  to  determine  the  optimum  number  and  location  of  electrodes,  the  best  way  to  collect  and 
process  individualized  net  training  data,  the  generalization  of  the  alertness  estimate  to  perfor¬ 
mance  on  more  complex  tasks,  and  ways  to  combine  more  than  one  channel  of  psychophysiolog- 
ical  information  to  make  the  alertness  estimates  more  accurate  and  noise-resistant.  Practical 
questions  regarding  easily  applied  electrode  montages  and  automatic  artifact  cancellation  also 
remain  to  be  investigated.  If  reliable  and  convenient  automatic  alertness  monitoring  can  be 
developed,  the  range  of  potentially  useful  applications  appears  to  be  wide. 
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